Virtual machine migration

ABSTRACT

A source virtual machine (VM) hosted on a source server is migrated to a destination VM on a destination server without first powering down the source VM. After optional pre-copying of the source VM&#39;s memory to the destination VM, the source VM is suspended and its non-memory state is transferred to the destination VM; the destination VM is then resumed from the transferred state. In one embodiment, the source VM memory is either paged into the destination VM on demand, or is transferred asynchronously by pre-copying and write-protecting the source VM memory, and then later transferring only the modified pages after the destination VM is resumed. In one embodiment, the source and destination servers share common storage, in which the source VM&#39;s virtual disk is stored; this avoids the need to transfer the virtual disk contents.

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 14/044,195 filed Oct. 2, 2013, which is acontinuation of U.S. patent application Ser. No. 13/599,759 filed Aug.30, 2012 (now U.S. Pat. No. 8,554,900), which is a continuation of U.S.patent application Ser. No. 12/721,899, filed Mar. 11, 2010 (now U.S.Pat. No. 8,260,904), which is a continuation of U.S. patent applicationSer. No. 12/352,242 filed Jan. 12, 2009 (now U.S. Pat. No. 7,680,919),which is a continuation of U.S. patent application Ser. No. 10/319,217,filed Dec. 12, 2002 (now U.S. Pat. No. 7,484,208), each of which isentitled “Virtual Machine Migration.” Each of the above applications isincorporated by reference in its entirety.

TECHNICAL FIELD

This invention relates to a computer architecture, in particular, to anarchitecture that coordinates the operation of multiple virtualmachines.

BACKGROUND

The advantages of virtual machine technology have become widelyrecognized. Among these advantages is the ability to run multiplevirtual machines on a single host platform. This makes better use thecapacity of the hardware, while still ensuring that each user enjoys thefeatures of a “complete,” isolated computer.

General virtualized computer system: As is well known in the field ofcomputer science, a virtual machine (VM) is a software abstraction—a“virtualization”—of an actual physical computer system. FIG. 1illustrates, in part, the general configuration of a virtual machine200, which is installed as a “guest” on a “host” hardware platform 100.

As FIG. 1 shows, the hardware platform 100 includes one or moreprocessors (CPU's) 110, system memory 130, and a storage device, whichwill typically be a disk 140. The system memory will typically be someform of high-speed RAM, whereas the disk (one or more) will typically bea non-volatile, mass storage device. The hardware 100 will also includeother conventional mechanisms such as a memory management unit MMU 150,various registers 160, and any conventional network connection device170 (such as a network adapter or network interface card—“NIC”) fortransfer of data between the various components of the system and anetwork 700, which may be any known public or proprietary local orwide-area network such as the Internet, an internal enterprise network,etc.

Each VM 200 will typically include at least one virtual CPU 210, avirtual disk 240, a virtual system memory 230, a guest operating system(which may simply be a copy of a conventional operating system) 220, andvarious virtual devices 230, in which case the guest operating system(“guest OS”) will include corresponding drivers 224. All of thecomponents of the VM may be implemented in software using knowntechniques to emulate the corresponding components of an actualcomputer.

If the VM is properly designed, then it will not be apparent to the userthat any applications 260 running within the VM are running indirectly,that is, via the guest OS and virtual processor. Applications 260running within the VM will act just as they would if run on a “real”computer, except for a decrease in running speed that will be noticeableonly in exceptionally time-critical applications. Executable files willbe accessed by the guest OS from the virtual disk or virtual memory,which will simply be portions of the actual physical disk or memoryallocated to that VM. Once an application is installed within the VM,the guest OS retrieves files from the virtual disk just as if they hadbeen pre-stored as the result of a conventional installation of theapplication. The design and operation of virtual machines is well knownin the field of computer science.

Some interface is usually required between a VM and the underlying hostplatform (in particular, the CPU), which is responsible for actuallyexecuting VM-issued instructions and transferring data to and from theactual memory and storage devices. A common term for this interface is a“virtual machine monitor” (VMM), shown as component 300. A VMM isusually a thin piece of software that runs directly on top of a host, ordirectly on the hardware, and virtualizes all the resources of themachine. Among other components, the VMM therefore usually includesdevice emulators 330, which may constitute the virtual devices (230)that the VM 200 addresses. The interface exported to the VM is then thesame as the hardware interface of the machine, so that the guest OScannot determine the presence of the VMM. The VMM also usually tracksand either forwards (to some form of operating system) or itselfschedules and handles all requests by its VM for machine resources, aswell as various faults and interrupts.

Although the VM (and thus the user of applications running in the VM)cannot usually detect the presence of the VMM, the VMM and the VM may beviewed as together forming a single virtual computer. They are shown inFIG. 1 as separate components for the sake of clarity.

Virtual and physical memory: As in most modern computers, the addressspace of the memory 130 is partitioned into pages (for example, in theIntel x86 architecture) or regions (for example, Intel IA-64architecture). Applications then address the memory 130 using virtualaddresses (VAs), which include virtual page numbers (VPNs). The VAs arethen mapped to physical addresses (PAs) that are used to address thephysical memory 130. (VAs and PAs have a common offset from a baseaddress, so that only the VPN needs to be converted into a correspondingPPN.) The concepts of VPNs and PPNs, as well as the way in which thedifferent page numbering schemes are implemented and used, are describedin many standard texts, such as “Computer Organization and Design: TheHardware/Software Interface,” by David A. Patterson and John L.Hennessy, Morgan Kaufmann Publishers, Inc., San Francisco, Calif., 1994,pp. 579-603 (chapter 7.4 “Virtual Memory”). Similar mappings are used inregion-based architectures or, indeed, in any architecture whererelocatability is possible.

An extra level of addressing indirection is typically implemented invirtualized systems in that a VPN issued by an application 260 in the VM200 is remapped twice in order to determine which page of the hardwarememory is intended. The first mapping is provided by a mapping modulewithin the guest OS 202, which translates the guest VPN (GVPN) into acorresponding guest PPN (GPPN) in the conventional manner. The guest OStherefore “believes” that it is directly addressing the actual hardwarememory, but in fact it is not.

Of course, a valid address to the actual hardware memory must ultimatelybe generated. A memory management module 350 in the VMM 300 thereforeperforms the second mapping by taking the GPPN issued by the guest OS220 and mapping it to a hardware (or “machine”) page number PPN that canbe used to address the hardware memory 130. This GPPN-to-PPN mapping istypically done in the main system-level software layer (such as thekernel 600 described below), depending on the implementation: From theperspective of the guest OS, the GVPN and GPPN might be virtual andphysical page numbers just as they would be if the guest OS were theonly OS in the system. From the perspective of the system software,however, the GPPN is a page number that is then mapped into the physicalmemory space of the hardware memory as a PPN.

System software configurations in virtualized systems: In some systems,such as the Workstation product of VMware, Inc., of Palo Alto, Calif.,the VMM is co-resident at system level with a host operating system.Both the VMM and the host OS can independently modify the state of thehost processor, but the VMM calls into the host OS via a driver and adedicated user-level application to have the host OS perform certain I/Ooperations of behalf of the VM. The virtual computer in thisconfiguration is thus fully hosted in that it runs on an existing hosthardware platform and together with an existing host OS.

In other implementations, a dedicated kernel takes the place of andperforms the conventional functions of the host OS, and virtualcomputers run on the kernel. FIG. 1 illustrates a kernel 600 that servesas the system software for several VM/VMM pairs 200/300, . . . , 200n/300 n. Compared with a system in which VMMs run directly on thehardware platform, use of a kernel offers greater modularity andfacilitates provision of services that extend across multiple VMs (forexample, for resource management). Compared with the hosted deployment,a kernel may offer greater performance because it can be co-developedwith the VMM and be optimized for the characteristics of a workloadconsisting of VMMs. The ESX Server product of VMware, Inc., has such aconfiguration.

A kernel-based virtualization system of the type illustrated in FIG. 1is described in U.S. patent application Ser. No. 09/877,378 (“ComputerConfiguration for Resource Management in Systems Including a VirtualMachine”), which is incorporated here by reference. The main componentsof this system and aspects of their interaction are, however, outlinedbelow.

At a boot-up time, an existing operating system 420 may be at systemlevel and the kernel 600 may not yet even be operational within thesystem. In such case, one of the functions of the OS 420 may be to makeit possible to load the kernel 600, after which the kernel runs on thenative hardware and manages system resources. In effect, the kernel,once loaded, displaces the OS 420. Thus, the kernel 600 may be viewedeither as displacing the OS 420 from the system level and taking thisplace itself, or as residing at a “sub-system level.” When interposedbetween the OS 420 and the hardware 100, the kernel 600 essentiallyturns the OS 420 into an “application,” which has access to systemresources only when allowed by the kernel 600. The kernel then schedulesthe OS 420 as if it were any other component that needs to use systemresources.

The OS 420 may also be included to allow applications unrelated tovirtualization to run; for example, a system administrator may need suchapplications to monitor the hardware 100 or to perform otheradministrative routines. The OS 420 may thus be viewed as a “console” OS(COS). In this case, the kernel 600 preferably also provides a remoteprocedure call (RPC) mechanism 614 to enable communication between, forexample, the VMM 300 and any applications 800 installed to run on theCOS 420.

Worlds: The kernel 600 handles not only the various VMM/VMs, but alsoany other applications running on the kernel, as well as the COS 420 andeven the hardware CPU(s) 110, as entities that can be separatelyscheduled. In this disclosure, each schedulable entity is referred to asa “world,” which contains a thread of control, an address space, machinememory, and handles to the various device objects that it is accessing.Worlds, represented in FIG. 1 within the kernel 600 as module 612, arestored in a portion of the memory space controlled by the kernel. Eachworld also has its own task structure, and usually also a data structurefor storing the hardware state currently associated with the respectiveworld.

There will usually be different types of worlds: 1) system worlds, whichare used for idle worlds, one per CPU, and a helper world that performstasks that need to be done asynchronously; 2) a console world, which isa special world that runs in the kernel and is associated with the COS420; and 3) virtual machine worlds.

Worlds preferably run at the most-privileged level (for example, in asystem with the Intel x86 architecture, this will be level CPL0), thatis, with full rights to invoke any privileged CPU operations. A VMM,which, along with its VM, constitutes a separate world, therefore mayuse these privileged instructions to allow it to run its associated VMso that it performs just like a corresponding “real” computer, even withrespect to privileged operations.

Switching worlds: When the world that is running on a particular CPU(which may be the only one) is preempted by or yields to another world,then a world switch has to occur. A world switch involves saving thecontext of the current world and restoring the context of the new worldsuch that the new world can begin executing where it left off the lasttime that it is was running.

The first part of the world switch procedure that is carried out by thekernel is that the current world's state is saved in a data structurethat is stored in the kernel's data area. Assuming the common case of anunderlying Intel x86 architecture, the state that is saved willtypically include: 1) the exception flags register; 2) general purposeregisters; 3) segment registers; 4) the instruction pointer (EIP)register; 5) the local descriptor table register; 6) the task register;7) debug registers; 8) control registers; 9) the interrupt descriptortable register; 10) the global descriptor table register; and 11) thefloating point state. Similar state information will need to be saved insystems with other hardware architectures.

After the state of the current world is saved, the state of the newworld can be restored. During the process of restoring the new world'sstate, no exceptions are allowed to take place because, if they did, thestate of the new world would be inconsistent upon restoration of thestate. The same state that was saved is therefore restored. The laststep in the world switch procedure is restoring the new world's codesegment and instruction pointer (EIP) registers.

When worlds are initially created, the saved state area for the world isinitialized to contain the proper information such that when the systemswitches to that world, then enough of its state is restored to enablethe world to start running. The EIP is therefore set to the address of aspecial world start function. Thus, when a running world switches to anew world that has never run before, the act of restoring the EIPregister will cause the world to begin executing in the world startfunction.

Switching from and to the COS world requires additional steps, which aredescribed in U.S. patent application Ser. No. 09/877,378, mentionedabove. Understanding this process is not necessary for understanding thepresent invention, however so further discussion is omitted.

Memory management in kernel-based system: The kernel 600 includes amemory management module 616 that manages all machine memory that is notallocated exclusively to the COS 420. When the kernel 600 is loaded, theinformation about the maximum amount of memory available on the machineis available to the kernel, as well as information about how much of itis being used by the COS. Part of the machine memory is used for thekernel 600 itself and the rest is used for the virtual machine worlds.

Virtual machine worlds use machine memory for two purposes. First,memory is used to back portions of each world's memory region, that is,to store code, data, stacks, etc., in the VMM page table. For example,the code and data for the VMM 300 is backed by machine memory allocatedby the kernel 600. Second, memory is used for the guest memory of thevirtual machine. The memory management module may include any algorithmsfor dynamically allocating memory among the different VM's 200.

Interrupt handling in kernel-based system: The kernel 600 preferablyalso includes an interrupt handler 650 that intercepts and handlesinterrupts for all devices on the machine. This includes devices such asthe mouse that are used exclusively by the COS. Depending on the type ofdevice, the kernel 600 will either handle the interrupt itself orforward the interrupt to the COS.

Device access in kernel-based system: In the preferred embodiment of theinvention, the kernel 600 is responsible for providing access to alldevices on the physical machine. In addition to other modules that thedesigner may choose to load into the kernel, the kernel will thereforetypically include conventional drivers as needed to control access todevices. Accordingly, FIG. 1 shows within the kernel 600 a module 610containing loadable kernel modules and drivers.

Kernel file system: In the ESX Server product of VMware, Inc., thekernel 600 includes a fast, simple file system, referred to here as theVM kernel file system (VMKFS), that has proven itself to be particularlyefficient for storing virtual disks 240, which typically comprise asmall number of large (at least 1 GB) files. By using very large filesystem blocks, the file system is able to keep the amount of metadata(that is, the data that indicates where data blocks are stored on disk)needed to access all of the data in a file to an arbitrarily small size.This allows all of the metadata to be cached in main memory so that allfile system reads and writes can be done without any extra metadatareads or writes.

The VMKFS in ESX Server takes up only a single disk partition. When itis created, it sets aside space for the file system descriptor, spacefor file descriptor information, including the file name, space forblock allocation information, and space for block pointer blocks. Thevast majority of the partition's space is used for data blocks, whosesize is set when the file system is created. The larger the partitionsize, the larger the block size should be in order to minimize the sizeof the metadata.

As mentioned earlier, the main advantage of the VMKFS is that it ensuresthat all metadata may be cached in high-speed, main system memory. Thiscan be done by using large data block sizes, with small block pointers.Since virtual disks are usually at least one gigabyte in size, usinglarge block sizes on the order of 64 Megabytes will cause virtually nowasted disk space and all metadata for the virtual disk can be cachedsimultaneously in system memory.

Besides being able to always keep file metadata cached in memory, theother key to high performance file I/O is to reduce the number ofmetadata updates. Note that the only reason why the VMKFS metadata willneed to be updated is if a file is created or destroyed, or if itchanges in size. Since these files are used primarily for virtual disks(or, for example, for copy-on-write redo logs), files are not oftencreated or destroyed. Moreover, because virtual disks are usually fixedin size upon creation, the file size of a virtual disk does not usuallychange. In order to reduce the number of metadata updates on a virtualdisk to zero, the system may therefore preallocate all data blocks forvirtual disks when the file is created.

Key VM features: For the purposes of understanding the advantages ofthis invention, the salient points of the discussion above are: (1) eachVM 200, . . . , 200 n has its own state and is an entity that canoperate completely independently of other VMs; (2) the user of a VM, inparticular, of an application running on the VM, will usually not beable to notice that the application is running on a VM (which isimplemented wholly as software) as opposed to a “real” computer; (3)assuming that different VMs have the same configuration and state, theuser will not know and would have no reason to care which VM he iscurrently using; (4) the entire state (including memory) of any VM isavailable to its respective VMM, and the entire state of any VM and ofany VMM is available to the kernel 600; (5) as a consequence of theabove facts, a VM is “relocatable.”

Except for the network 700, the entire multi-VM system shown in FIG. 1can be implemented in a single physical machine, such as a server. Thisis illustrated by the single functional boundary 1000. (Of coursedevices such as keyboards, monitors, etc., will also be included toallow users to access and use the system, possibly via the network 700;these are not shown merely for the sake of simplicity.)

In systems configured as in FIG. 1, the focus is on managing theresources of a single physical machine: Virtual machines are installedon a single hardware platform and the CPU(s), network, memory, and diskresources for that machine are managed by the kernel 600 or similarserver software. This represents a limitation that is becomingincreasingly undesirable and increasingly unnecessary. For example, ifthe server 1000 needs to be shut down for maintenance, then the VMsloaded in the server will become inaccessible and therefore useless tothose who need them. Moreover, since the VMs must share the singlephysical memory space 130 and the cycles of the single (or single groupof) CPU, these resources are substantially “zero-sum,” such thatparticularly memory- or processor-intensive tasks may cause noticeablyworse performance.

One way to overcome this problem would be to provide multiple servers,each with a set of VMs. Before shutting down one server, its VMs couldbe powered down or checkpointed and then restored on another server. Theproblem with this solution is that it still disrupts on-going VM use,and even a delay of ten seconds may be noticeable and irritating tousers; delays on the order of minutes will normally be whollyunacceptable.

What is needed is a system that allows greater flexibility in thedeployment and use of VMs, but with as little disruption to users aspossible. This invention provides such a system, as well as a relatedmethod of operation.

SUMMARY

In a networked system of computers (preferably, servers), including asource computer and a destination computer and a source virtual machine(VM) installed on the source computer, the invention provides avirtualization method and system according to which the source VM ismigrated to a destination VM while the source VM is still powered on.

Different methods are provided for transferring the source VM's memoryto the destination VM. In the preferred embodiment of the invention, thedestination VM may be resumed before transfer of the source VM memory iscompleted. One way to do this is to page in the source VM memory to thedestination VM on demand. Following an alternative procedure, the sourceVM memory is pre-copied to the destination VM before the non-memorysource VM state information is transferred.

In one refinement of the invention, any units (such as pages) of thesource VM memory that are modified (by the source VM or by any othercomponent) during the interval between pre-copying and completingtransfer of all pages are retransferred to the destination VM.Modification may be detected in different ways, preferably bywrite-protecting the source VM memory and then sensing page faults whenthe source VM attempts to write to any of the protected memory units. Aniterative procedure for retransferring modified memory units is alsodisclosed.

In the preferred embodiment of the invention, the source VM's non-memorystate information includes the contents of a source virtual disk. Thecontents of the source virtual disk are preferably stored in a storagearrangement shared by both the source and destination computers. Thedestination VM's virtual disk is then prepared by mapping the virtualdisk of the destination VM to the same physical addresses as the sourcevirtual disk in the shared storage arrangement.

In the most commonly anticipated implementation of the invention,communication between a user and the source and destination VMs takesplace over a network. Network connectivity is preferably also madetransparent to the user by arranging the servers on a common subnet,with virtual network connection addresses generated from a common namespace of physical addresses.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the main components of a server that includes one ormore virtual machines running on a system-level kernel.

FIG. 2 illustrates a farm of interconnected servers according to theinvention, with each server hosting a plurality of virtual machines.

FIG. 3 illustrates the steps the invention takes to migrate a virtualmachine from a source to a destination.

DETAILED DESCRIPTION

In broadest terms, the invention provides a farm of servers, each ofwhich may host one or more virtual machines (VMs), as well as mechanismsfor migrating a VM from one server (the source server) to another (thedestination server) while the VM is still running. There are manyreasons why efficient, substantially transparent VM migration isbeneficial. Load-balancing is mentioned above, as is the possibilitythat a machine may need to be taken out of service for maintenance.

Another reason may be to add or remove resources from the server. Thisneed not be related to the requirements of the hardware itself, butrather it may also be to meet the desires of a particular user/customer.For example, a particular user may request (and perhaps pay for) morememory, more CPU time, etc., all of which may necessitate migration ofhis VM to a different server.

The general configuration of the server farm according to the inventionis illustrated in FIG. 2, in which a plurality of users 900, 902, 904, .. . , 906 access a farm of servers 1000, 1002, . . . , 1004 via thenetwork 700. Each of the servers is preferably configured as the server1000 shown in FIG. 1, and will include at least one and possibly manyVMs. Thus, server 1000 is shown in both FIG. 1 and FIG. 2.

In a server farm, all of the resources of all of the machines in thefarm can be aggregated into one common resource pool. From theperspective of a user, the farm will appear to be one big machine withlots of resources. As is described below, the invention provides amechanism for managing the set of resources so that they are utilized asefficiently and as reliably as possible.

Advantages of VM migration: The ability to quickly migrate VMs whilethey are running between individual nodes of the server farm has severaladvantages, among which are the following: (1) It allows the load to bebalanced across all nodes in the cluster. If one node is out ofresources while other nodes have free resources, then VMs can be movedaround between nodes to balance the load. (2) It allows individual nodesof the cluster to be shut down for maintenance without requiring thatthe VMs that are running on the node be shut down: Instead of shuttingthe VMs down, the VMs can simply be migrated to other machines in thecluster. (3) It allows the immediate utilization of new nodes as theyare added to the cluster. Currently running VMs can be migrated frommachines that are over utilized to newly added machines that have freeresources.

Goals of VM migration: There are two primary goals for VM migration:First, the time that the migrated VM is unavailable should be minimized.This is because if the VM is unavailable for more than a few seconds,clients that depend on services exported by the VM may become unhappy.The migration therefore needs to be transparent to clients of the VM.Second, the time that the VM is dependent on state stored on the sourcemachine should also be minimized because as long as the VM is dependenton the source machine the VM is less fault-tolerant than before it wasmigrated.

Assumptions: The preferred embodiment of the invention makes twoassumptions, which, although not necessary to the main definition of theinvention, greatly simplify its implementation. Those skilled in the artof computer design and system-level programming will be able to adaptthe preferred embodiment of the invention to avoid the need for theseassumptions, for example, to provide a custom implementation forparticularly important users. The two simplifying assumptions are:First, VMs will be migrated only between machines that share the storagewhere the VMs' disks reside. This assumption eliminates the need tomigrate entire disks. In order to allow for inter-server migration, theservers 1000, 1002, . . . , 1004 therefore either share an external,common storage system or can access each other's internal storage. Oneway to arrange this is for all the servers in the farm to be connectedvia a system such as Fibrechannel; this is illustrated in FIG. 2 as thechannel 1700. Second, in the preferred embodiment of the invention, VMsare migrated only between machines whose network adapters are on thesame subnet that is being used by the VM.

Suspend-and-resume: Existing VM technology makes possible the suspensionand resumption of a VM. This feature is found, for example, in the ESXServer product mentioned above. Moreover, co-pending U.S. patentapplication Ser. No. 09/497,978, filed 4 Feb. 2000 (“EncapsulatedComputer System”), which is incorporated here by reference, discloses amechanism for checkpointing the entire state of a VM. When a VM issuspended, all of its state (including its memory) is written to a fileon disk. A VM can then be migrated by suspending the VM on one serverand resuming it via shared storage on another server. Unfortunately,writing out the saved state, especially the memory, to disk and thenreading it back in again on the new server is too slow. This isespecially true for VMs with large memories. Even a relatively small 512Mbyte VM would, for example, take at least 20 or 30 seconds to suspendand then resume again. Larger VMs of 2 or 4 Gigabytes would take evenlonger. These delays are unacceptable.

Another disadvantage of migration using the current technology is thatit consumes large amounts of disk bandwidth, especially if thesuspend-and-resume operations must be done quickly. This can beespecially problematic if the reason that a VM is being migrated isbecause the machine (server) that it is running on is low on availabledisk bandwidth.

Note that the execution of a VM is frequently suspended even though itis “running.” A VM may be suspended, for example, in order to allowexecution of another co-running VM to proceed. Suspending the VM longenough to transfer its non-memory state is therefore not inconsistentwith the notion that it is still running. Suspension for the purpose ofnon-memory state transfer contrasts however, with powering down or“shutting off” the VM, which is the software mechanism that virtualizesthe power-down procedure of a physical machine. For example, suspensiondoes not necessarily lead to loss of cached data, whereas powering-offtypically does. Similarly, resumption of execution after a suspensiondoes not require such time-consuming tasks as rebooting the OS, whereaspowering back on (“restarting”) typically does. The invention allows thesource VM to be migrated with no need to power down the VM.

Migration: FIG. 2 illustrates migration of a VM (the source VM 1200) inserver 1000 to a VM (the destination VM 1202) in server 1002. Therespective servers may therefore be referred to respectively as the“source” and “destination” servers or machines. In order to migrate a VMfrom one server (the source) to another (the destination), three thingsneed to be transferred to the destination machine. In decreasing orderof size, these are 1) the disk; 2) the memory; and 3) saved non-memorystate for the frame buffer, all virtual devices and whatever else isneeded to resume from a checkpoint. This additional required informationwill depend on how the VMs are configured, but will be known to thedesigner who wishes to take advantage of this invention.

Note that memory itself is included in the concept of the state of a VM;indeed, memory is the dominant portion. In the preferred embodiment ofthe invention, however, memory is treated as a special case because itis relatively so large.

Now recall the first assumption above that the servers on which thesource and destination VMs run share storage. This means that the sourceVM's virtual disk can be transferred without copying any data—all thatis needed is for the destination VMM to ensure that the mappings for thedestination VM's virtual disk are to the same physical addresses in theshared storage arrangement as the source VM's, which will not havechanged. The other state, however, must be transferred to thedestination machine in order to complete the migration. According to theinvention, this other state information is transferred directly betweenthe source and destination machines, without performing the intermediatestep of writing the state to disk.

Memory is either transferred ahead of time while the VM is running or ispaged in from the source machine after the VM has resumed on thedestination machine. All other saved non-memory state is transferreddirectly from the source VM to the destination machine. The only timethe VM will be unavailable to the user is therefore while the savednon-memory state is being transferred by the source to the destinationand while the saved non-memory state is being restored on thedestination. Because the saved non-memory state is typically less than20 Mbytes, and because this transfer can be direct, that is, requiringno intermediate physical disk operations, it will take only a fewseconds to save and restore the non-memory state across a conventionalbroadband network link.

In FIGS. 2 and 3, the VMMs for each VM are not shown simply for the sakeof simplicity. The VM can be transferred without requiring any transferof state information about its respective VMM, and the VMM istransparent to the VM. Nonetheless, for the sake of simplicity, when itis stated below that the VM performs a certain migration-related action,it is to be understood that the instructions for performing theseactions are preferably not part of the VM's code (in order to maintaintransparency). Rather, the migration code may be located in any ofseveral components that are external and transparent to the VM beingmigrated: In a prototype of the invention, for example, the migrationcode was included within the VMM (module 360), which interacted with adedicated user-level application 860 configured as in the VMwareWorkstation product mentioned above.

Migration Components

Migration Components: There are several entities involved in performinga migration, each of which is a software module, that is, a set ofcomputer instructions, that will execute on the processor(s) of therespective server:

1. An initiator of the migration, which may be located anywhere in thesystem other than in the VM to be migrated. A load-balancing module insome supervisory system that monitors the operation of the servers mightinitiate migration, for example. An administrator (human and/orautomatic) might also initiate migration of VMs from one server to oneor more others because of a need for maintenance, repair, upgrade, etc.As another example, a kernel will generally be aware of the memoryavailable to the VMs it supports; if these VMs' memory requirements areparticularly high, then the kernel might poll other kernels to find adifferent server whose memory load is currently less and then migrateone or more VMs to that server. In one prototype of the invention, themigration initiator was formed of cooperating software modules withinthe VMM of the VM to be migrated and the dedicated user-levelapplication 860 (FIG. 1). In FIG. 3, the initiator is shown as beinglocated in a general migration sub-system 2000, which will be includedin whatever component(s) the designer prefers for a given implementationof the invention.

2. A service process or “daemon” 1300 that runs on the destinationserver and that is capable of configuring and starting up thedestination VM. Such daemons are found in existing virtualizationsystems, such as those of VMware, Inc.

3. The source VM 1200 for the migration. This is the VM that is runningon the source server 1000 and that is being migrated to the destinationserver 1002.

4. The destination VM 1202 for the migration.

5. The kernel (or equivalent system software) on the destination machine(the “destination kernel” 1602).

6. The kernel (or equivalent system software) on the source machine (the“source kernel” 1600).

Migration Procedures: According to the invention, the main stepsinvolved in a migration are: a) preparing for VM migration; b)optionally, pre-copying at least a portion of the source VM memory tothe destination; c) suspending the source VM while transferring itsnon-memory state information to the destination VM; and d) transferringremaining memory from the source to the destination. Each of these mainsteps in turn comprises several sub-steps, which are illustrated in FIG.3.

FIG. 3 is a structural flowchart (showing both the operations andcomponents involved) of the procedures according to the preferredembodiment of the invention for migrating a VM. In FIG. 3, the variousprocedural steps are numbered 1-13. As mentioned above, no VM needs toinclude code to perform any of the actions involved in the migration;rather, this code will be located in the VM's respective VMM or someother user- and/or system-level software component(s). In FIG. 3,various migration steps are shown within the source and destination VMsmerely for the sake of clarity and to illustrate that it is thesecomponents that are the object of the operations.

Preparing for migration: The first step in a VM migration is that theinitiating sub-system (within component 2000) requests the migration(Step 1) for any reason, such as those mentioned as examples above. Inorder to minimize the down time for a running source VM 1200 during themigration process, a destination VM 1202 needs to be created and putinto a wait state before the source VM begins migration. Upon receipt ofany suitable initiation signal and parameters from the sub-system 2000,the destination VM 1202 is therefore created (Step 2) by a service—theserver daemon 1300—that runs on the destination machine. As is mentionedabove, such VM creation routines are found in existing virtualizationsystems; any such known routine may be used in the invention.

Once the destination VM is created, its VMM informs the daemon 1300 thatit is ready to accept migration (Step 3). The daemon then calls into thedestination kernel 1602 to notify it that state information can betransferred from the source VM. The destination kernel will return anerror if no state is available to be restored. The VM expects this and,as part of waiting for migration (Step 3A), keeps retrying until thedestination kernel is ready to begin restoring state. The daemon 1300also indicates to other components (for example, the source anddestination kernels and/or the general migration component 2000) thatthe destination VM is ready for migration (Step 4).

Transferring non-memory state information: Once the destination VM isready, the source VM is suspended long enough (using known techniques)that its non-memory state information can be transferred to and saved inthe destination VM. The sub-system 2000 therefore indicates to thesource VMM and/or kernel 1600 that the source VM should be suspended andthat migration should be started (Step 5). Note that a non-memory statetransfer will usually be a relatively fast operation compared withtransferring the source VMs memory, so that the source VM's suspensionshould not be so long that it inconveniences the user.

Transferring non-memory state information: The source VM 1200 (that is,its VMM) first interacts with the destination kernel 1602 to signal thekernel to prepare to accept the saved non-memory state. Once executionof the source VM is suspended, the source VM's non-memory state can besaved, that is, transferred and stored in the destination (Step 6).Transfer may be done using any known technology, such as over a TCP/IPconnection.

If asynchronous memory transfer is implemented (see below), the sourceVM's memory may also be pre-copied (Step 7) to the destination machinewhile the source VM is still running. The source memory is dealt withspecially and is discussed below. Once all the source VM's non-memorystate information has been saved (Step 8) in the destination, the sourceVM (again, meaning its VMM) sends a message to the destination kernelinforming it that non-memory state transfer has completed (Step 9) andthat the destination VM can now begin restoring itself (Step 10), thatis, loading the saved non-memory state information and executing thedestination VM from the restored state. When the non-memory stateinformation is fully restored within the destination VM (Step 11), thedestination VM will be functionally identical to the suspended sourceVM, except for the source VM's memory.

Dealing with memory: The physical memory of the VM (that is, thecontents of the virtual memory 230, which the VM views as physicalmemory) is dealt with differently than all other non-memory state. Thisis because memory is too large to transfer all at once after the VM hassuspended execution. Even over a gigabit link, it will typically take aminimum of 10 seconds per gigabyte of data transferred. A 4 gigabyte VMwould therefore have to be suspended for 40 or more seconds, and evenlonger if the gigabit link is not dedicated to the migration. As aresult, according to the invention, the memory is either transferredwhile the source VM is still running or the destination machine pages itin after the destination VM starts running.

The invention provides for different ways to migrate the source VM'smemory, both of which involve paging in from the source VM's memory(Step 12), either on demand or asynchronously, any pages not alreadycopied to the destination, until all the source VM's memory pages havebeen transferred to the destination (Step 13). One method is for thedestination machine to demand page in all memory over the network afterthe destination VM has started. In other words, a page of memory fromthe source VM is transferred to the destination VM when the destinationVM needs and requests it. This requires a minimal implementation effortbut has two disadvantages: First, the destination VM may experience poorperformance when it is paging in its physical memory over the network;and second, the destination VM will “die” if the source machine crashesor is rebooted. The second problem can be mitigated to some extent byincluding a background paging thread that pages in all pages from thesource machine. This will limit the amount of time that the destinationmachine is dependent on the source machine.

Following a more sophisticated method according to the invention, asmuch memory as possible is transferred asynchronously while the sourceVM is running. This is done as follows:

1) A thread is created, preferably in the source VM's VMM, whose job itis to push, that is, pre-copy, all of the source VM's memory over to thedestination machine.

2) The thread iterates through all physical pages and does thefollowing: (a) The PPN associated with the physical page iswrite-protected by the source VMM, using known procedures, which may beincluded in the memory management module 350; and (b) The thread writesthe page over to the destination kernel.

If any of the pages that were transferred are modified (the VMM detectsthis because a write-protect fault will be taken on the page), then thepage is marked as modified. Marking may be done as simply—andcompactly—as setting a bit in a table (or vector) for the modified page.Note that the first time the destination VM touches any page that hasnot been copied to the destination machine, it will require a networkpage fault to the source machine; this fault is then used as a signal toimmediately transfer the needed page. Once all pages have beentransferred, a list of modified pages is then sent to the destinationkernel so it knows that these pages need to be paged in from the source.

According to the invention, it would also be possible to pre-copy thesource VM's memory to the destination “iteratively,” that is, overmultiple “passes,” before the non-memory state is transferred. Accordingto this iterative memory transfer procedure, a first set (preferablyall) of pages of the source VM's memory is pre-copied to the destinationVM. During the time the memory is being transferred, however, the sourceVM, which is allowed to continue running, may modify some of thetransferred pages. Modifications to the source VM's memory may bedetected and tracked using any known method, such as a write-protectionmechanism or a separate table indicating memory modifications. Thesemodified pages are then re-transferred to the destination. While theyare being transferred, however, the source VM may modify other pages (oreven modify a previously modified page again). The newly modified pagesare then retransferred, and so on.

The system repeats the iterative memory transfer procedure until thenumber of pages left to be transferred is less than some threshold orthe system notices that no forward progress is being made (no reductionin the number of newly modified pages still to be transferred). Thethreshold (which may be zero), may be determined as an absolute orrelative number of pages, either ahead of time, or according to anyknown adaptive routine.

Each subsequent iteration should take less time because fewer pages willneed to be transferred; the transfer process should therefore convergetowards a number of modified pages that is small enough that they cantransferred rapidly. Any newly modified pages remaining to be copiedover after the threshold has been reached may then be transferred afterthe source VM is suspended and before non-memory state is copied over;alternatively, these remaining pages may be paged in by the destinationVM either on demand or asynchronously after the source VM is suspendedand the destination VM is resumed from the suspended source state.

The iterative memory transfer method is perhaps best understood byconsidering an example: For the initial memory transfer, assume that4096 pages are pre-copied from the source VM to the destination VM, butthat the source VM then modifies 1024 of these pages. During a secondtransfer iteration, the 1024 pages are transferred while the source VMis still running. Assume, however, that during or just after this seconditeration, the source VM modifies not only 200 of the 1024 pages butalso 100 of the original 3072 (=4096-1024). There will then be 300modified pages still not transferred. Assume that 300 is less than thechosen threshold. Once the source VM is suspended, these 300 pages canbe transferred either before or after the non-memory state istransferred. If they are transferred before the destination VM isresumed, then they are preferably transferred all at once. If they aretransferred after the destination VM is resumed, then they mayalternatively be paged in either on demand or asynchronously.

Depending on the number of pages that are modified by the source VMwhile the preparation phase is being executed (see above), thispre-copying solution (iterative or not) may significantly reduce thenumber of network page faults needed after the destination VM isresumed. This will in turn improve performance as well as shorten thetime required for the background paging thread to get all of the memoryover to the destination machine.

The pre-paging approach has a couple of disadvantages, however. First,it increases the time it takes to completely migrate a VM—the VM cannotbe migrated until all of its memory has been copied over to thedestination machine. Second, it requires that more memory be transferredthan the first approach—any pages that are modified after they aretransferred will have to be transferred twice. The designer of anyimplementation of the invention may decide which method—demand paging orpre-paging or a hybrid of both—to include by following known designconsiderations.

Once all the state, including the memory, is transferred and thedestination VM is resumed, the source VM, or rather, an exact copy ofthe source VM, will be installed in the destination server and willfunction exactly as the source VM did.

Networking: Recall the assumption, for the preferred embodiment of theinvention, that the source and destination kernels are attached to thesame subnets. This allows the new VM to have the same networkconnectivity as before. Other methods may also be used to providenetwork connectivity, such as packet forwarding.

A Media Access Control (MAC) address is the well known, standard,conventional hardware address that uniquely identifies each node of anetwork. Similar node addressing schemes are found in other networks.The MAC address of each virtual NIC is preferably generated out of thename space of MAC addresses that is owned by the administrator of theserver farm system. The MAC address will then not depend on the MACaddress of the physical NIC.

Except for a short delay, the user will not be aware that he is nowcommunicating with a different server than before. As a result, once theVM is restored on the destination machine, users of the VM will beentirely unaware that the VM has moved, so that even networking will beable to continue to work flawlessly.

Cleaning up: The source machine can free all resources (includingmemory) once the last page of memory has been fetched from it andtransferred to the destination. This clean-up may be done using knownroutines, and may be the same as those (with obvious modifications) nowused to uninstall VMs.

Disk transfer: As is mentioned above, one assumption made in thepreferred embodiment is that the source and destination servers sharethe storage where the source and destination VMs' disks reside. Thisarrangement speeds up the transfer process (often greatly) since iteliminates the need to migrate entire disks, which can be very large. Asthe speed of processors, network connections, and storage devicesincreases, this assumption may no longer be necessary in order to ensuresmooth, substantially unnoticeable VM migration.

Even with existing technology, the invention does not strictly require acommon server storage system as long as the additional delay of virtualdisk transfer is more acceptable to the VM user than the inconvenienceand delay of powering down a running VM. A user might want to transferhis running VM to another remote machine, for example, perhaps foranother user to continue using, or for diagnostic purposes, withoutlosing any cached data.

In such cases, the virtual disk may be transferred using the sametechniques as are described above for memory transfer, that is, usingon-demand and/or asynchronous page (or sector or track, etc.) transferfrom the course VM's virtual disk to the destination VM's virtual disk.In addition, the pre-copying techniques used for memory transfer areapplicable to disk transfer as well—the disk data can be transferredwhile the VM is still running and any modified disk blocks (or similarunits) can then be fetched by the destination VM after it is restored,either all at once or iteratively.

What is claimed is:
 1. A computer-readable medium having stored thereininstructions which, when executed by a processor, perform a method ofmigrating a source virtual machine (VM) running on a source computer toa destination computer, the method comprising: suspending execution ofthe source VM; after the suspending of the source VM, transferringnon-memory state of the source VM to the destination computer, therebyenabling execution of a destination VM on the destination computer fromthe transferred non-memory state of the source VM; and transferringcontents of source VM memory to the destination computer, either whilethe source VM is executing, or after the execution of the destination VMfrom the transferred non-memory state of the source VM.
 2. Thecomputer-readable medium of claim 1, the method further comprising:executing the destination VM on the destination computer from thetransferred non-memory state.
 3. The computer-readable medium of claim2, the method further comprising: creating the destination VM beforesuspending the source VM; and loading the transferred non-memory stateof the source VM into the destination VM.
 4. The computer-readablemedium of claim 2, wherein the contents of the source VM memory aretransferred to the destination computer after the execution of thedestination VM from the transferred non-memory state of the source VM.5. The computer-readable medium of claim 2, wherein the contents of thesource VM memory are transferred to the destination computer while thesource VM is executing, and the method further comprising: after thesuspending of the source VM, retransferring to the destination computercontents of any units of the source VM memory that were modified sincecontents of the source VM memory were previously transferred from thesource computer to the destination computer.
 6. The computer-readablemedium of claim 5, wherein the step of retransferring to the destinationcomputer is performed before the execution of the destination VM fromthe transferred non-memory state of the source VM.
 7. Thecomputer-readable medium of claim 5, wherein the step of retransferringto the destination computer is performed after the execution of thedestination VM from the transferred non-memory state of the source VM.8. The computer-readable medium of claim 2, wherein the step oftransferring contents of source VM memory to the destination computercomprises: iteratively transferring the contents of the source VM memoryto the destination computer while the source VM is executing, whereiniteratively transferring includes retransferring to the destinationcomputer any units of the source VM memory that were modified since thesource VM memory was previously transferred from the source computer tothe destination computer.
 9. The computer-readable medium of claim 2,wherein the source VM's non-memory state does not include contents of asource virtual disk.
 10. The computer-readable medium of claim 9, themethod further including: storing the contents of a source virtual diskin a storage arrangement shared by both the source and destinationcomputers; and preparing a virtual disk of the destination VM by mappingthe virtual disk of the destination VM to the source virtual disk in theshared storage arrangement.
 11. The computer-readable medium of claim 9,the method further including: transferring contents of the sourcevirtual disk on a source storage device to a destination storage deviceof the destination computer, either while the source VM is executing, orafter the execution of the destination VM from the transferrednon-memory state of the source VM.
 12. The computer-readable medium ofclaim 11, the method further including: wherein the contents of thesource virtual disk are transferred to the destination storage deviceafter the execution of the destination VM from the transferrednon-memory state of the source VM.
 13. The computer-readable medium ofclaim 11, wherein the contents of the source virtual disk aretransferred to the destination storage device while the source VM isexecuting, and the method further comprising: after the suspending ofthe source VM, transferring to the destination storage device contentsof any disk blocks of the source virtual disk that were modified oradded since contents of the source virtual disk were previouslytransferred from the source virtual disk to the destination storagedevice.
 14. A computer system comprising: a source computer having asource virtual machine (VM) running thereon; a destination computerconfigured to support one or more VMs running thereon; and computersoftware executing on the source and destination computers andimplementing a method for migrating the source VM to the destinationcomputer, the method comprising: suspending execution of the source VM;after the suspending of the source VM, transferring non-memory state ofthe source VM to the destination computer; executing the destination VMon the destination computer from the transferred non-memory state; andtransferring contents of source VM memory to the destination computer,either while the source VM is executing, or after the execution of thedestination VM from the transferred non-memory state of the source VM.15. The computer system of claim 14, wherein the contents of the sourceVM memory are transferred to the destination computer after theexecution of the destination VM from the transferred non-memory state ofthe source VM.
 16. The computer system of claim 14, wherein the contentsof the source VM memory are transferred to the destination computerwhile the source VM is executing, and the method further comprising:after the suspending of the source VM, retransferring to the destinationcomputer contents of any units of the source VM memory that weremodified since contents of the source VM memory were previouslytransferred from the source computer to the destination computer. 17.The computer system of claim 14, wherein the step of transferringcontents of source VM memory to the destination computer comprises:iteratively transferring the contents of the source VM memory to thedestination computer while the source VM is executing, whereiniteratively transferring includes retransferring to the destinationcomputer any units of the source VM memory that were modified since thesource VM memory was previously transferred from the source computer tothe destination computer.
 18. The computer system of claim 17, whereinthe retransferring of the units of the source VM memory that weremodified is executed a plurality of times.
 19. The computer system ofclaim 18, wherein each subsequent retransferring involves a lessernumber of the units than a preceding retransferring.
 20. The computersystem of claim 19, wherein retransferring of the units of the source VMmemory that were modified is repeated until the number of the units isless than a threshold value, and then transferring the number of unitsafter suspending the source VM.
 21. The computer system of claim 14,wherein the source VM's non-memory state does not include contents of asource virtual disk.
 22. The computer system of claim 21, the methodfurther including: transferring contents of the source virtual disk on asource storage device to a destination storage device of the destinationcomputer, either while the source VM is executing, or after theexecution of the destination VM from the transferred non-memory state ofthe source VM.
 23. The computer system of claim 21, the method furtherincluding: storing the contents of a source virtual disk in a storagearrangement shared by both the source and destination computers; andpreparing a virtual disk of the destination VM by mapping the virtualdisk of the destination VM to the source virtual disk in the sharedstorage arrangement.
 24. A method of migrating a source virtual machine(VM) running on a source computer to a destination computer, the methodcomprising: suspending execution of the source VM; after the suspendingof the source VM, transferring non-memory state of the source VM to thedestination computer, thereby enabling execution of a destination VM onthe destination computer from the transferred non-memory state of thesource VM; and transferring contents of source VM memory to thedestination computer, either while the source VM is executing, or afterthe execution of the destination VM from the transferred non-memorystate of the source VM.
 25. The method of claim 24, the method furthercomprising: executing the destination VM on the destination computerfrom the transferred non-memory state.
 26. The method of claim 25, themethod further comprising: creating the destination VM before suspendingthe source VM; and loading the transferred non-memory state of thesource VM into the destination VM.
 27. The method of claim 24, whereinthe contents of the source VM memory are transferred to the destinationcomputer after the execution of the destination VM from the transferrednon-memory state of the source VM.
 28. The method of claim 27, whereinthe transferring of contents of the source VM memory comprises paging inone or more of the units of the source VM memory on demand.
 29. Themethod of claim 27, wherein the transferring of contents of the sourceVM memory comprises paging in one or more of the units of the source VMmemory asynchronously.
 30. The method of claim 27, wherein none of unitsof the source VM memory is copied to the destination computer before theexecution of the destination VM from the transferred non-memory state.31. The method of claim 24, wherein the source VM's non-memory statedoes not include contents of a source virtual disk.
 32. The method ofclaim 31, the method further including: transferring contents of thesource virtual disk on a source storage device to a destination storagedevice of the destination computer, either while the source VM isexecuting, or after the execution of the destination VM from thetransferred non-memory state of the source VM.
 33. The method of claim31, the method further including: storing the contents of a sourcevirtual disk in a storage arrangement shared by both the source anddestination computers; and preparing a virtual disk of the destinationVM by mapping the virtual disk of the destination VM to the sourcevirtual disk in the shared storage arrangement.